Problems With the ACAOM Second Resolution and Recommendations

Why did ACAOM decide at its August 2009 meeting to revisit a
February 9, 2008 resolution on the development and piloting of accreditation
standards for accrediting first-professional doctorate programs in acupuncture
and in Oriental medicine, as stated in its Second Resolution on
First-Professional Doctoral Standards?Why did it adopt a motion that would “continue the comment
period for seeking information and consensus regarding the first professional
doctorate until January 15th, 2010”?Based on what comments and from what stakeholders?How was this decision reached, or more
correctly, why was this resolution passed?

 

The February 2008 Resolution On First-Professional Doctorate Standards by ACAOM succinctly laid
down the background of the evolution of the first-professional
doctorate.  The net result of
this process was the declaration by ACAOM that there was at that time
“insufficient evidence of consensus within the acupuncture and Oriental
communities of interest…to warrant implementing a first-professional doctorate
as entry-level into the profession.”ACAOM resolved that the AOM community should continue to seek consensus
“by whatever means deemed appropriate regarding the issue of a
first-professional doctorate as entry-level into the profession” and that once
such consensus is reached, “the Commission will renew its efforts to develop
and subsequently pilot standards, policies and procedures for
first-professional doctoral programs in AOM as entry-level into the
profession.”So this leads to the
important, but as yet, unconsidered questions regarding the FPD posed in the
opening paragraph.What comments
did ACAOM receive, from whom were they from and what qualified this input as
sufficient to reopen the comment period?It is my contention that there was a sufficient lack of evidence from
the practitioner, student and educational communities to reopen the comment
period that we are in now.I
assert that if the evidence I suspect is behind the ACAOM motion to continue comment on the FPD, then ACAOM should repeal its motion to continue the comment period regarding the FPD as this “evidence” is
borderline disingenuous in its presentation and has major sampling issues that
make the data contained therein unreliable as a gauge of support for the FPD.

 

First we must consider the population that composes the
stakeholder community.The U.S.
Secretary of Education defines this community in 34 CFR 602.13 as “educators
and educational institutions, licensing bodies, practitioners and employers in
the professional…fields for which the educational institutions or programs
within the agency’s jurisdiction prepare their students.” According to a letter
written in early 2009 by Mark Seem, the president and founder of the Tri-State
College of Acupuncture and a vocal proponent of the FPD, “…the overall number
of practitioner respondents to AAAOM’s December 2008 survey on the FPD is far
higher, and represents significantly greater support for the FPD for entrance
into the AOM field, than ACAOM’s initial survey in 2003.”
Seem argues that the “survey provides sufficient
evidence of consensus from the practitioner community for ACAOM to renew its
review of the draft FPD standards.”Further, in addition to a Council of Colleges motion in May of 2008
which supported the development of appropriate FPD standards and supported a
renewal of ACAOM’s review of those standards, Seem posits that there is
sufficient evidence from all stakeholders for ACAOM to do just that.This letter/position paper by Seem appears to address ACAOM directly, likely intended for the ACAOM Commissioners.It exists through the Dragon Rises
website, accessible by a Google search.The date is in the title of the document Google created during a web crawl, which dates its composition at
1.11.09.The ACAOM Commissioners
met in Columbia, Maryland for their winter Strategic Planning Session over the
weekend of February 6-8th.It is possible that the ACAOM considered Seem’s
position, but decided to wait until the AAAOM survey was complete before
considering reopening the comment period.Indeed, Seem states that the preliminary data he is privy to is “sufficient for ACAOM to renew its review of the draft FPD standards so that appropriate FPD standards might be developed and adopted in a timely fashion” and he asks that “ACAOM consider the adoption of these draft standards now, so that they might enter the trial criteria process and in order for colleges that so choose to begin the final processes necessary for piloting these FPD programs.”  The AAAOM survey ended on May 13th.

 

Following the chronology of the FPD timeline, it is
reasonable to postulate that the ACAOM Second Resolution on First
Professional Doctorate Standards in August of 2009 was adopted as a direct
result of the line of reasoning outlined by Seem in his January position
paper.The ACAOM resolution
states, “Based on comments received from a number of stakeholders in response
to the Commission-adopted resolution, ACAOM adopted a motion at its summer 2009
meeting to continue the comment period for seeking information and consensus
regarding the first professional doctorate until January 15th,
2010.”This is, of course, where
we are at now.And up until this
point, no one has questioned exactly what “comments” were received and from
which “stakeholders”.And that is
what I am asking here.Because if
the commentary is essentially the AAAOM survey conducted from November of 2008
until May of 2009, then the ACAOM must consider immediately rescinding the
Second Resolution that reopened the comment period.The AAAOM survey is problematic in several critical aspects.It has an inferior sampling methodology
and takes liberties in its data presentation.It also does not show that there is “significantly greater
support for the FPD as entrance into the AOM field, than ACAOM’s initial survey
in 2003” as Seem proclaims.Lacking this, and fundamental flaws in its sampling method, the comment
period should have never been reopened.At the very least, the information presented here must be considered by
the ACAOM Commissioners at their winter meeting in February and in a fashion
similar to its 2008 resolution, they should table the doctorate once again.

 

AAAOM SURVEY

 

First off, the way that the AAAOM presents their survey
information up front is borderline disingenuous.Summarizing the survey data, the report states, “For the FPD
in acupuncture, of those who expressed an opinion 65% were supportive and 38% were not
supportive.”Immediately, we have
a major problem with the way the data is being presented.The only
way a
substantial majority (65%) of respondents “support” the FPD is if you consider
a subgroup of the total sample: “those who expressed an opinion”.What does this mean?Not only do the figures presented at
the outset leave out those that had “No Opinion”, they also leave out the
people who happened to be neutral or undecided- the “Neithers”.If one looks at Appendix A, they will see very quickly how
the real numbers add up.55%, as
opposed to 65%, actually support the FPD in acupuncture from all survey categories-
administrators, students and practitioners.There is a majority, but a slimmer and less impressive
one.Now remember, this is for all
categories and Dr. Seem was claiming that there is significantly greater
support for the FPD from practitioners with the AAAOM survey than with the 2003
ACAOM survey.In that survey,
38.2% of practitioners showed any support for the doctorate, though it appears
that the survey is assessing support for both acupuncture and for Oriental
medicine, not just acupuncture.And indeed, the AAAOM survey summary reports that 57% of practitioners
showed any support for the FPD in acupuncture.That’s a leap
of nearly 20%.However, the
numbers being displayed up front are only of those who expressed an
opinion.
When including the “Neither” and “No Opinion” data,
the real number is 48% practitioner support, less than an overall majority.This means that the difference is
actually only about 10% from the ACAOM 2003 survey.This increase is much less “significant.”In short, a majority of
practitioners surveyed by the AAAOM did not support the FPD in acupuncture.
All the categories can be assessed this
way and all the support numbers come down from what is presented in the page 2
summary for all respondent categories, for both acupuncture as well as for
Oriental medicine.

 

The real numbers do not indicate the support that the survey
summary cheerily reports for the doctorate in acupuncture or in Oriental
medicine.  The finer distinctions of this data were not made very clear by the
hard as nails reporting of Acupuncture Today.Yet,
even with weaker support than advertised by AT, the AAAOM and presumably reported
to ACAOM at their Winter and Summer 2009 meetings, the real problem isn’t with
these numbers, it is with the data, or more specifically, how it was gathered.

The AAAOM survey, and likely its ACAOM
predecessor, is a non-random sample.It was primarily distributed through e-mails, internet forum posts, and
website mentions.What this means,
is that there is no way to determine whether those that completed the survey
actually represent general opinion in the field.
The survey is on the
small side, with 662 practitioners responding out of nearly 30,000. But again, the real problem is that the
survey is not randomized properly in a probability sampling scheme.With a probability sampling scheme
“is one in which every unit in the population has a chance (greater than
zero) of being selected in the sample, and this probability can be
accurately determined. The combination of these traits makes it
possible to produce unbiased estimates of population totals, by
weighting sampled units according to their probability of selection.”  Examples include Simple Random Sampling, Systematic Sampling, Stratified Sampling,
Probability Proportional to Size Sampling, and Cluster or Multistage
Sampling; any could have been used to accurately determine the presence or lack thereof of support for the FPD amongst stakeholders. 

A
truly indicative sample of support from the AOM practitioner community would
have the names of all AOM practitioners and select a random person from that population.These selected
individuals would then be interviewed.This kind of sampling can be expensive and time consuming, but gives a more accurate assessment of the population.So, it is impossible to generalize the 662 practitioner responses in the AAAOM survey to the
nearly 30,000 licensed practitioners in the US.
Hence, the AAAOM survey should be taken with a large grain
of salt- treated cautiously and not taken too seriously.But it appears that the ACAOM may have
done just the opposite and reopened the comment period on what amounts to an
internet survey.And while it is
conceded that random sampling is expensive and time consuming, so is the
doctorate degree that this survey is opening up the door to.And accordingly, the resources need to
be gathered by whatever means are appropriate to determine actual support for
the doctorate from a random sample of the entire AOM community before the
standards may continue to be drafted and programs may be piloted.But again, this needed to happen before
the comment period reopened.In
any case, ACAOM must consider the claims of the AAAOM survey as being
representative of general practitioner opinion as baseless and cease the
drafting of standards and piloting of the doctorate.Of course, the same can be said for all categories of
opinion in the survey.Random
sampling should be done for students and administrators as well.Until this happens, any claim of
opinion of entire populations of stakeholders should be seen for what they are-
speculative and potentially subject to bias.

 

As a matter of fact, there is a high likelihood that the
both practitioner and student opinion represented in the AAAOM survey is
overrepresented by the Tri-State College of Acupuncture student body and
alumni.In Seem’s letter to ACAOM,
he states that the “AAAOM survey developers were unsuccessful in soliciting
cooperation from AOM colleges to help AOM students complete this survey,
probably because this request came when most AOM colleges were in exam week or
on winter break already.”Yet,
Tri-State conducted a survey of its own student and alumni population from
November 18th to 28th. The AAAOM survey commenced on
November 21st. The Tri-State survey gathered 83 student responses
and 148 practitioner responses.The total response count for students on the AAAOM survey was 208.Practitioners were tallied at 662.It is reasonable to theorize that AAAOM
survey announcements were able to find their way to the Tri-State student body,
being that their own survey started at the same time and likely did not
coincide with exams, as well as the obvious strong opinions of the schoolspresident, Dr. Seem.As a result,
there is a strong possibility that the opinion of the students and
alumni/practitioners of Tri-State are heavily overrepresented in the AAAOM
survey.This phenomenon is known
as selection bias.Wikipedia defines
selection bias as a “statistical bias in which there is an error in choosing
the individuals or groups to take part in a scientific study.It is sometimes referred to as the
selection effect.The term
“selection bias” most often refers to the distortion of a statistical analysis,
resulting from the method of collecting samples.If the selection bias is not taken into account
then any conclusions drawn may be wrong.”
(emphasis mine)If the survey was done in a fashion
that is acceptable to statisticians, this information could be either confirmed
or refuted.It is assumed that the
ACAOM has the names of all respondents to check for this error.But even if it wasn’t the case and
Tri-State was not accounting for a disproportionately higher volume of AAAOM
survey responses, this survey would still be non-random and likely not
representative of the greater AOM population, whether the opinion of students,
practitioners or administrators.Even still, the Tri-State FPD survey is also non-random.

 

As a result of the manner in which the survey was
distributed, there are other characteristics that may be shared by those that
completed the survey that aren’t necessarily representative of the entire field
of practitioners or students for that matter.These characteristics may include; recent graduates, people
that own a computer, people with a current e-mail address, people that frequent
online forums, those that participate in online discussion boards, and AAAOM
members.These categories are
speculative, though based on the distribution strategies they are
reasonable.It is great that the
survey designers stratified by AAAOM membership, however random sampling was not done within each strata. Also,
it seems that many acupuncturists do not participate in online discussion
boards, especially those that do not practice for whatever reason.These groups would be underrepresented
in the population sample and may have strong feelings about the issue.

 

These speculations are not meant to be presented as a matter
of bad faith on the survey designers, or Dr. Seem for that matter.Nor is the allegation that ACAOM may have reumed the comment period over faulty data meant to give them a black eye.  Those who designed the survey likely
did everything in their power to get a higher response rate.The problem isn’t the survey questions or the
survey designers.The problem is
with the methodology and the ensuing data.

 

The AAAOM survey analysis is therefore not rooted in
solid data.The survey may suggest
“majority support for FPDs by the AOM profession”, but it does so only if one
discounts the most basic elements of sound sampling methodology.Any results and interpretation of the
AAAOM survey data and ensuing comparison to the ACAOM 2003 survey data are
highly speculative and are should not have been used as evidence for ACAOM to
reopen the comment period for the FPD.It is strongly suggested that the ACAOM rescind its
Second Resolution if the AAAOM survey data is the primary reason for continuing
the comment period of the FPD.As
the data is highly questionable due to sampling issues, it cannot serve the
function ascribed to it by Mark Seem; it is not indicative of a significant
show of approval from students or practitioners of AOM.If this is not the data that lead to a
continuance of the commentary period concerning the FPD, then it is the
responsibility of the ACAOM to reveal exactly what stakeholder information
reopened the comment period.It is
also strongly suggested that the AOM profession go about conducting a census of
the profession through the appropriate professional organization(s).A proper random
sample of the licensed acupuncturist population is necessary to make inferences about the
opinions of the larger group.Without this step, any further action taken by the ACAOM allowing for
the development of doctoral draft standards and subsequent pilot programs will
never be able to demonstrate that these procedures are ‘widely accepted” by the
AOM stakeholder community, and therefore short of the requirements in USDE 34
CFR 602.13.

*much appreciation to both Dr. Kim Tippens of the Helfgott Research Institute and an anonymous disenchanted PhD student on the verge of leaving academia for a life of travel writing using my questions to procrastinate on her never-going-to-happen dissertation with guidance and clarity on these issues.

Related Articles

Responses

Your email address will not be published. Required fields are marked *

  1. AOM Community Forum postings: “We are owed an explanation”

    Hmmm…I’m so kung fu tze (confucious, confused).

    The FPD powers that be seem to not fully grasp what it is they’re trying to accomplish either.

    It would seem (ode to you know who…haha) that the discussion on the Professional Doctorate (not a “first” professional doctorate to ms hobbs) is being muddled with semantics. Thanks to Matt Bauer for demanding clarification.

    to sum it up:
    “Entry into the profession could happen at either the Masters or Doctorate
    level. ” (jabbour).

    Mr. Jabbour even sought clarification for ACAOM with regards to semantics surrounding the Professional Doctorate as its intention was not to “replace the entry level masters” and is not, in fact, a “first” professional doctorate.

    I see the veils are coming down, and it really is about ego stroking. So what is it really? And why have to different, yet “seemingly equal” entry points to unlimited licensure (or is there something dubious in future state legislation that may come of it?

    Respect from other health professions? Not so much if the “powers that be” cain’t (intentional misspelling) even figgerit out.

    Going back to Jabbours initial interpretation that entry to the trade (cuz that’s what it is–regional vs national accreditation speak) could happen at the masters or the doctorate.

    To put it in inner city hospital unit vernacular: “Honey, that don’t make no kinda sense atall!”

  2. this post is a thing of beauty

    not least because in the midst of this incredibly muddled discussion, I also managed to lose track of the extremely important question you just put back into the center, where it belongs:

    Wait, why are we talking about this again? No, really — WHY ARE WE TALKING ABOUT THIS AGAIN? 

    Thank you Larry, thank you Kim, thank you anonymous Ph.D student.

    “You know how people always say there’s a reasonable explanation for things like this? Well, there isn’t.” Daniel Pinkwater, The Neddiad